What is @csstools/css-tokenizer?
@csstools/css-tokenizer is a library for tokenizing CSS. It breaks down CSS code into a stream of tokens, which can be useful for building CSS parsers, linters, or other tools that need to analyze or manipulate CSS code.
What are @csstools/css-tokenizer's main functionalities?
Tokenizing CSS
This feature allows you to tokenize a CSS string. The `tokenize` function takes a CSS string and returns an iterable of tokens, which can then be processed or analyzed.
const { tokenize } = require('@csstools/css-tokenizer');
const css = 'body { color: red; }';
const tokens = tokenize(css);
for (const token of tokens) {
console.log(token);
}
Handling different token types
This feature demonstrates how to handle different types of tokens. The tokens returned by the `tokenize` function are arrays where the first element is the type of token and the second element is the value. This allows you to differentiate between different token types such as words, whitespace, etc.
const { tokenize } = require('@csstools/css-tokenizer');
const css = 'body { color: red; }';
const tokens = tokenize(css);
for (const token of tokens) {
if (token[0] === 'word') {
console.log('Word token:', token[1]);
} else if (token[0] === 'whitespace') {
console.log('Whitespace token');
}
}
Other packages similar to @csstools/css-tokenizer
postcss
PostCSS is a tool for transforming CSS with JavaScript plugins. It includes a tokenizer and parser, making it a more comprehensive tool compared to @csstools/css-tokenizer, which focuses solely on tokenization.
css-tree
CSSTree is a toolset for CSS including a fast detailed parser, walker, generator, and lexer. It provides more advanced features like AST generation and manipulation, whereas @csstools/css-tokenizer focuses on tokenization.
css
The 'css' package is a CSS parser and stringifier. It provides a full parser that converts CSS into an AST and back, offering more functionality than just tokenization, which is the primary focus of @csstools/css-tokenizer.
CSS Tokenizer
![Discord](https://shields.io/badge/Discord-5865F2?logo=discord&logoColor=white)
Implemented from : https://www.w3.org/TR/2021/CRD-css-syntax-3-20211224/
API
Read the API docs
Usage
Add CSS Tokenizer to your project:
npm install @csstools/css-tokenizer --save-dev
import { tokenizer, TokenType } from '@csstools/css-tokenizer';
const myCSS = `@media only screen and (min-width: 768rem) {
.foo {
content: 'Some content!' !important;
}
}
`;
const t = tokenizer({
css: myCSS,
});
while (true) {
const token = t.nextToken();
if (token[0] === TokenType.EOF) {
break;
}
console.log(token);
}
Or use the tokenize
helper function:
import { tokenize } from '@csstools/css-tokenizer';
const myCSS = `@media only screen and (min-width: 768rem) {
.foo {
content: 'Some content!' !important;
}
}
`;
const tokens = tokenize({
css: myCSS,
});
console.log(tokens);
Options
{
onParseError?: (error: ParseError) => void
}
onParseError
The tokenizer is forgiving and won't stop when a parse error is encountered.
To receive parsing error information you can set a callback.
import { tokenizer, TokenType } from '@csstools/css-tokenizer';
const t = tokenizer({
css: '\\',
}, { onParseError: (err) => console.warn(err) });
while (true) {
const token = t.nextToken();
if (token[0] === TokenType.EOF) {
break;
}
}
Parser errors will try to inform you where in the tokenizer logic the error happened.
This tells you what kind of error occurred.
Goals and non-goals
Things this package aims to be:
- specification compliant CSS tokenizer
- a reliable low level package to be used in CSS parsers
What it is not: